Feature selection - one form of dimension reduction

Even if \(X\) full rank, we might want eliminate some predictors to decrease model variance.

• near - zero variance predictors

• highly correlated predictors (with each other)

• predictors uncorrelated with the response

• Forward/backward stepwise selection
• Lasso regression
• …

Curse of dimensionality - simple analytical example

Suppose KNN with \(p\) predictors \(\sim U(0,1)\)

• volume of hypercube with sides of length \(d\) is \(d^p\).

• To capture \(r\) proportion of the data, we want \(d^p = r\), so \(d = r^{1/p}\)

Features

• predictors or variables used in a model

• transformed and combined versions of the original predictors

Feature engineering is the art of creating useful features; it may increase or decrease the number of predictors.

Dimension reduction

Reducing the number of features/dimension of feature space

• Feature selection - select a subset of the original predictors

• Feature extraction - create new predictors from the original ones

  • PCA, UMAP, tSNE, neural networks/deep learning

Principal Components Analysis (PCA)

Create features - components from linear combinations of the predictors

• The first component is in the direction of maximum variance of the data*.

• Second component in direction of max variance orthogonal (and this uncorrelated) to the first, and so on.

PCA - the math

Predictors \(X_1, X_2, \dots, X_p\); form linear combinations

\[ Z_m = \sum_{j=1}^p \phi_{jm} X_j, \qquad m = 1, 2, \dots, p \]

In matrix form,

\[ Z = X \mathbf{U} \]

Now with data

\[ \mathbf{Z} = \mathbf{XU} \]

Principal components and eigenvectors

First scale or standardize the data matrix.

The principal components (columns of \(\mathbf{U}\)) are the eigenvectors of \(\mathbf{X}^T \mathbf{X}\).

(Equivalently, the eigenvectors of the correlation matrix - and dont need to standardize.)

Theorem

• Eigenvectors of \(\mathbf{X}^T \mathbf{X}\) are orthogonal

• Let \[ \lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p > 0 \] with eigenvectors \(\mathbf{v}_1, \dots, \mathbf{v}_p\)

  • \(\lambda_i\) is variance in direction \(\mathbf{v}_i\)
  • \(\mathbf{v}_1\) maximizes variance
  • \(\mathbf{v}_2\) maximizes variance subject to orthogonality with \(\mathbf{v}_1\)
  • and so on

Principal components regression (PCR) in practice

1. Center and scale the data

2. Form \(\mathbf{U}\), the eigenvectors of the centered and scaled \(\mathbf{X}^T \mathbf{X}\), ordered by decreasing eigenvalues

3. Compute \(\mathbf{Z} = \mathbf{XU}\)

4. Select the number of components based on explained variance (or cross-validation)

5. Fit a linear regression using the first \(m\) components

Example: meatspec data

Analytical chemistry - predict fat content of meat samples from 100 channel infrared absorbance spectra.

PCA components cumulative variances

 [1] 0.98626 0.99596 0.99875 0.99990 0.99996 0.99999 0.99999 1.00000 1.00000
[10] 1.00000

R squared vs RMSE

Source: Tidy Modeling with R, Max Kuhn and Julia Silge, 2023

Example: wine data (UCI ML Repository)

Chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the **three types* of wines.

Total and explained variance of the PCA components

Multinomial regression on the PCA features

We get perfect score with just 5 components, and 98% with only 3 components.

Loadings and scores

 [1] 0.361988 0.554063 0.665300 0.735990 0.801623 0.850981 0.893368 0.920175
 [9] 0.942397 0.961697 0.979066 0.992048 1.000000

Visualizing the loadings

Just the first PCA component

Visualizing the loadings

First four PCAcomponents

Example: USArrests dataset¹ (

Loadings and scores

[1] 0.620060 0.867502 0.956642 1.000000

Visualizing the feature contributions in the components

Biplots

We can visualize the loadings and the scores together with a biplot.

UMAP: penguins revisited

PCA vs UMAP on penguins

Penguins pairplot

UMAP may be overkill here

UMAP

mnist dataset¹ - images of handwritten digits (0-9). 784 features (28 x 28 pixels)

PCA vs UMAP on mnist

Feature extraction

Transforming and combining predictors to form new “features”